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(54) A person identifier and management system 

(57) A person identifier and management system 
(10) includesaface image database (13) that stores the 
face feature data of each of a number of face images 
and the person-identifying data associated with each of 
the face images. A camera (23) is also included in the 
person identifier and management system (10) for cap- 
turing an input face image. A face analysis system (12) 
is then coupled to the camera (23) and the image data- 



base (13) to extract the face feature data of the input 
face image and to compare the face feature of the input 
face image with that of each of the face images stored 
in the database (13) such that the person-identifying 
data of any stored face image similar to the input face 
image can be retrieved from the image database (13). 
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Description 

[0001] The present invention pertains to image 
processing and information management. More particu- 
larly, this invention relates to a person identifier and 
management system that can retrieve person-identify- 
ing information from a database based on an input face 
such that the input face can be identified. 
[0002] As is known, human nature typically shows that 
people want to be recognized by others. Thus, it is 
important for people to recognize and recall the names 
of other people they have previously met or acquainted 
during meetings, social functions, or other occasions. 
Not recognizing the person before you may affect you 
adversely. For example, if a superior or employer does 
not recognize or cannot recall the name of an inferior or 
employee, it may give the inferior/employee the impres- 
sion that the supervisor/employer does not care about 
the inferior/employee, or the inferior/employee is not 
important in the organization. In addition, if a business 
person cannot recognize or recall the name of a busi- 
ness counterpart from another organization, it may 
affect the business dealing or potential business dealing 
with the organization that business counterpart repre- 
sents. 

[0003] However, for people who have a relatively large 
number of casual acquaintances, association of faces 
with names is a difficult memory game. In addition, 
some people are just not good at recognizing or recall- 
ing names of formal acquaintances, especially when the 
formal acquaintances have only been briefly introduced 
to you. Change of environment may also cause prob- 
lems for people to recognize or recall names of others. 
For example, a person will have more difficulty in recog- 
nizing people initially if the person is surrounded by a 
large number of strangers (e.g., in a new office build- 
ing). 

[0004] With the advance of computer and image 
processing technologies, it is now possible to store an 
image in a computer system. This can be done using a 
digital camera to capture an image. In addition, a scan- 
ner can be used to convert a printed image into digital 
form The digital image includes image pixels (or dots) 
arranged in a matrix. The data for each image pixel are 
then stored in the computer. The digital image can be 
retrieved for display from the computer. 
[0005] This allows a face image database to be built 
in a computer system that can help the user to manage 
the acquaintances' information. The face image data- 
base typically stores a number of face images and the 
person-identifying information associated with each of 
the stored images. The person-identifying information is 
typically a text information that identifies the face. The 
person-identifying information in text format may include 
name, contact information, date and occasion of last 
meeting or all the prior meetings. When the user 
retrieves a face image from the database, the person- 
identifying information associated with that image is 
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also retrieved from the database. When the person- 
identifying information is displayed with the associated 
face image, the user can then recognize the face and 
recall the name of the face. 

5 [0006] However, searches within this face image data- 
base is still nearly impossible. In order to identify a 
desired face image stored in the computer based on an 
input face image, the prior art typically causes the com- 
puter to compare the image pixel data of the input face 

w image with the corresponding pixel data of each of the 
images stored in the computer. The comparison typi- 
cally takes a very long time to complete and uses a lot 
of system resources (e.g., memory and communication 
bandwidth). This is because the computer has to com- 

rs pare each image pixel data of the input face image with 
the corresponding image pixel data of each of the 
stored face images. A match occurs when each pixel 
data of the input face image is similar the corresponding 
pixel data of a stored face image. Using this method, 

so unless the two compared images are extremely similar, 
it is very difficult to find a match between any two face 
images. This is especially true because noise and other 
positional errors may be introduced into the image data 
when the input face image is captured. 

2S [0007] One feature of the present invention is to help 
a user recall or recognize people the user has previ- 
ously met 

[0008] Another feature of the present invention is to 
allow retrieval of personal-identifying information from a 

so database in accordance with an input face image. 
[0009] A further feature of the present invention is to 
provide a person identifier and management system 
that searches for similar faces of an input face in a face 
image database such that person-identifying informa- 

35 Son associated with the similar faces can be retrieved 
from the database to identify the input face. 
[001 0] A still further feature of the present invention is 
to provide a personal person identifier and management 
system that is portable, personal, and easy to use. 

« [00111 A person identifier and management system is 
described. The person identifier and management sys- 
tem includes an image database that stores face feature 
data of each of a number of face images and person- 
identifying data associated with each of the face 

45 images. An image capturing system is also included for 
capturing an input face image. A face analysis system is 
then coupled to the image capturing system and the 
image database to locate and retrieve the person-iden- 
tifying data of any stored face image similar to the input 

so face image from the image database. The face analysis 
system does this by, for example, extracting the face 
feature data of the input face image and comparing the 
face feature data of the input face image with that of 
each of the face images stored in the database. 

55 [001 2] A method of retrieving person-identifying infor- 
mation from a face image database based on an input 
face image includes the step of extracting face feature 
data from the input face image. The face feature data of 
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the input lace image is then compared with that of a 
number of face images stored in the database. Each of 
the stored face images includes the person-identifying 
information that identifies the respective face image. If 
the face feature data of one of the stored face images 
matches that of the input face image, then the person- 
identifying information of that stored face image is 
retrieved from the image database to identify the input 
face image. 

[0013] Other features and advantages of the present 
invention will become apparent from the following 
detailed description, taken in conjunction with the 
accompanying drawings, illustrating by way of example 
the principles of the invention. 

Figure 1 shows the structure of a person identifier 
and management system that implements one 
embodiment of the present invention. 
Figure 2 illustrates the perspective view of the per- 
son identifier and management system of Figure 1 
as a portable, personal, and hand-held system in 
accordance with one embodiment of the present 

Figure 3A is a front view of the personal person 
identifier and management system of Figure 2. 
Figure 3B shows the top view of the personal per- 
son identifier and management system of Figure 2. 
Figure 4 shows the analysis system of the person 
identifier and management system of Figure 1 . 
Figure 5 shows the input/output system of the per- 
son identifier and management system of Figure 1. 
Figure 6 shows the data format for each of the face 
images stored in the profile database of the person 
identifier and management system of Figure 1. 

[0014] Figure 1 shows a person identifier and man- 
agement system 10 that implements one embodiment 
of the present invention. As will be described in more 
detail below, the person identifier and management sys- 
tem 10 allows a user to retrieve person-identifying or 
face-identifying information of a person from a profile 
database (or face image database) 13 of the person 
identifier and management system 10 based on the per- 
son's face inputted to the person identifier and manage- 
ment system 10 such that the person can be identified 
or recognized by the user of the person 'identifying sys- 
tem 10. This means that the person identifier and man- 
agement system 10 can use the input face image to 
search the profile database 13 for matching face or 
faces and their associated face-identifying or person- 
identifying data so that the input face can be identified 
or recognized. 

[001 5] As will be described in more detail below, the 
person identifier and management system 10 includes 
the profile database 13. The profile database 13 stores 
a number of face images and their associated face- 
identifying or person-identifying data. The profile data- 
base 1 3 stores the pixel data of each of the face images. 



In addition, the profile database 13 also stores the face 
feature data of each of the face images. The face fea- 
ture data of a face image is associated with that face 
image in the profile database 1 3. The person-identifying 

5 data of each of the face images is also associated with 
the respective face image in the profile database 13. 
[001 6] In addition to the profile database 1 3, the per- 
son identifier and management system 10 also includes 
an analysis system 12 that (1) detects if an input image 

10 of the person identifier and management system 10 
contains a face image and (2) extracts the face feature 
data from the input image if the input image is detected 
to contain a face. The input image is supplied by an 
input/output system 11 connected to the analysis sys- 

15 tern 1 2. The input/output system 1 1 is also used for out- 
putting the person-identifying data retrieved from the 
profile database 13. 

[0017] The analysis system 12 accesses the profile 
database 13 with the extracted face feature data of the 

20 input face image to locate the matching face feature 
data in the profile database 13. When there is a match 
between the face feature data of the input face image 
and the face feature data of one or more stored face 
images, the person-identifying data of the matching 

zs face images are retrieved from the profile database 1 3. 
This allows the user of the person identifier and man- 
agement system 10 to identify or recognize the input 
face image. The person identifier and management sys- 
tem 10 will be described in more detail below, also in 

30 conjunction with Figures 1 -6. 

[001 8] Figure 1 shows in block diagram form the struc- 
ture of the person identifier and management system 
10. The person identifier and management system 10 
can be a single stand-alone device or implemented in a 

35 computer or computer-enabled system. If the person 
identifier and management system 10 Is implemented in 
a computer system, the computer system can be a per- 
sonal computer, a notebook computer, a mini-computer, 
a hand-held computer, a personal communicator, a 

w workstation, a network computer, a server computer, or 
a mainframe computer. 

[001 9] As can be seen from Figure 1 , the person iden- 
tifier and management system 10 includes the 
input/output system 1 1 , the analysis system 1 2, and the 
45 profile database 13. The input/output system 1 1 is con- 
nected to the analysis system 12. The analysis system 
12 is then connected to the profile database 13. The 
input/output system 11 is used provide the input and 
output functions for the person identifier and manage- 
so ment system 10. The analysis system 12 is used to 
detect a face from an input image and to use the face 
feature data extracted from the input face image to 
search the profile database 13 in order to identify the 
input face image. 
55 [0020] The profile database 13 stores a number of 
face images and their face feature data and person- 
identifying or face-identifying data. The face feature 
data and the person-identifying data are fully searcha- 
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We in the database 13. The face feature data and its 
creation will be described in more detail below, also in 
conjunction with Figures 1 and 4. The person-identifying 
data can be a text data, an image data, and a combina- 
tion of text and image data. The person-identifying data 
of a face image may include the name of the person, 
affiliation or employment of the person, and time and 
occasion of last meeting or all of the previous meetings 
with the person. The employment information may 
include name of the company the person is currently 
working for, the title and position of the person within the 
company. In addition, the person-identifying data may 
also include other information, such as telephone and 
fax numbers and/or e-mail address, the name of the 
third person who introduced the person, and/or other 
types of person-identifying information. 
[0021] Each of the digital face images stored in the 
profile database 13 is, for example, in the pixel matrix 
format. This means the profile database 13 stores the 
pixel data of each of the stored face images. In addition, 
the face feature data and the person-identifying data of 
each face Image are associated with or linked to the 
respective face image stored in the profile database 1 3. 
The face feature data and the person-identifying data of 
a face image are, however, not necessarily physically 
stored adjacent to the respective face image in the pro- 
file database 13. The profile database 13 can be imple- 
mented and accessed using known database 
technologies. Figure 6 shows the logical memory map 
100 for the face images stored in the profPe database 
13. 

[00221 As can be seen from Figure 6, the logical mem- 
ory map 100 of the profile database 13 of Figure 1 
includes a number of entries 101 through I00n, each for 
storing one digital face image. Each entry (e.g., the 
entry 101) includes a face feature data field (e.g., the 
face feature data field 101a), a person-identifying data 
field (e.g.. the person-identifying data field 101b), and 
the image pixel data field (e.g., the pixel data field 101c). 
The pixel data field of each entry stores the pixel data 
(i.e.. the image data) of a face image stored in the pro- 
file database 13. The face feature data field of each 
entry stores the face feature data of a face image stored 
in the database 13 and the person-identifying data field 
of each entry stores the person-identifying data of a 
face image stored into the database 13. The face fea- 
ture data field of an entry is associated with or linked to 
the face image of that entry. The person-identifying data 
field of an entry is also associated with or linked to the 
face feature data of that entry. 
[0023] Referring back to Figure 1, the input/output 
system 11 is used to input face images to the person 
identifier and management system 10. In addition, the 
input/output system 11 is also the output means of the 
person identifier and management system 10 for data 
output. For example, the input/output system 1 1 outputs 
the person-identifying data of a matching face stored in 
the person identifier and management system 10 by 



displaying or printing the person-identifying data. The 
input/output system 1 1 may include a number of input 
and output devices or modules. For example, the 
input/output system 1 1 may include a display that dis- 
5 plays the input face image to be searched in the profile 
database 13 for matching faces. The display also dis- 
plays any person-identifying data retrieved from the pro- 
file database 13. In addition, the input/output system 1 1 
may also include a digital camera that captures the 
io input face image. The digital camera may be replaced, 
for example, with a scanner. Moreover, the input/output 
system 11 may include a text input device (e.g., key- 
board or touch sensitive screen with handwriting recog- 
nition capabilities) that allows input of text person- 
is identifying data into the profile database 13. Figure S 
shows one implementation of the input/output system 
11 of Figure 1, which will be described in more detail 
below. 

[0024] As can be seen from Figure 5, the input/output 
X system 1 1 includes an I/O controller 51 connected to a 
number of IAD devices 52 through 56. The function of 
the I/O controller 51 is to control various input/output 
activities of the input/output system 1 1 . The I/O control- 
ler 51 is implemented using known I/O controller tech- 
25 nologies. 

[0025] The I/O devices 52-56 include a camera 52, a 
display 53, a scanner 54, a sensitive pen 55, and other 
I/O devices 56. The camera 52 can be any known digital 
camera The display 53 can be any known computer or 

30 TV display. In one embodiment, the display 53 is a touch 
sensitive screen display that allows text and/or graphics 
input using a sensitive pen. In this case, the touch sen- 
sitive screen display may have an embedded keyboard 
or may have a handwriting recognition system. All of the 

as I/O devices 52-55 can be implemented using known 
technologies. 

[0026] The other I/O devices 56 may include a key- 
board, a printer, and a modem or network adaptor for 
network communications. All the other I/O devices 56 
40 are known I/O devices. Although Figure 5 show I/O 
devices 52-56 for the input/output system 11, the 
input/output system 1 1 may include more or fewer than 
I/O devices 52-56. For example, the input/output system 

1 1 may not include the scanner 54. 

4s [0027] Referring back to Figure 1, the analysis system 

1 2 of the person identifier and management system 1 0 
is used to detect the face from the input face image sup- 
plied from the input/output system 11. In addition, the 
analysis system 12 extracts face feature data from the 

so detected face in the input face image. The analysis sys- 
tem 12 then uses the extracted face feature data to 
search the profile database 1 3 for matching face feature 
data stored. If matching face feature data is located in 
the profile database 13, then the analysis system 12 

ss retrieves the person-identifying data associated with the 
matching face feature data stored in the profile data- 
base 13. When the user of the person identifier and 
management system 10 views the retrieved person- 
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identifying data, the user is able to recognize or identify 
the input face image. 

[0028] If no matching face feature data is found in the 
profile database 13, H means that none of the stored 
face images in the profile database 13 is similar to the 
input face image. In this case, the analysis system 12 
makes such indication to the user. The user can then 
have the option of storing the input face image in the 
profile database 13. along with the associated face fea- 
ture data and the person-identifying data. Before storing 
the input face image into the profile database 13, the 
analysis system 12 asks the user to input the person- 
identifying data of the input face image via the input/out- 
put system 1 1 . The structure and function of the analy- 
sis system 12 will be described in more detail below, 
also in conjunction with Figure 4. 
[0029] Referring to Figure 4, the analysis system 12 
includes a user interface 41 , a search engine 42, and a 
face detection and feature extraction system 43. The 
user interface 41 is connected to the search engine 42 
and the face detection and feature extraction system 43. 
The search engine 42 is also connected to the face 
detection and feature extraction system 43. Each of the 
components 41-43 is a functional module and can be 
implemented by software, hardware, and firmware. 
[0030] The user interface 41 is an interactive user 
interface that is part of an interactive window interface. 
The user interface 41 includes a window (not shown) 
that displays the input face image and the retrieved per- 
son-identifying data. In addition, the user interface 41 
also includes a number of command or function buttons 
that can be activated using a user input device (e.g., 
mouse or sensitive pen). The user interface 41 may also 
include a touch sensitive screen keyboard or a hand 
writing recognition system. The user interface 41 can be 
implemented using any known technology. 
[0031] The face detection and feature extraction sys- 
tem 43 receives the input face image from the user 
interface 41. The face detection and feature extraction 
system 43 detects if the input face image contains any 
face. The detection system 43 also detects the eye loca- 
tions of the face in the input face image. The system 43 
then uses the eye location data to compute the face fea- 
ture data of the detected face. The face feature data are 
also referred to as face coefficient data. 
[0032] In one embodiment, the detection system 43 
uses neural network-based face detection technology to 
detect the face in the input face image. The neural net- 
work-based face detection scheme is disclosed in a 
publication entitled HUMAN FACE DETECTION IN VIS- 
UAL SCENES, by H. Rowley (har@cs.cmu.edu), S. Bal- 
uja (baluja@cs.cmu.edu), and T. Kanade 
(tk@cs.cmu.edu) in November 1995. The publication is 
available from Carnegie Mellon University's Internet site 
at %»ww.ius.cs.cmu.edu/tUS/h8^ar/www/CMU-CS- 
95-1 58R/." Alternatively, other known face detection 
technologies may be used. For example, a face detec- 
tion scheme described in U.S. Pat No. 5.164,992. dated 



November 17. 1992. and entitled FACE RECOGNITION 
SYSTEM can be employed. 

[00331 When the detection system 43 uses the neural 
network-based face detection scheme, the detection 

5 system 43 detects if the input face image contains any 
face by first applying a set of neural network-based fil- 
ters to the image. The filters examine each location in 
the image at several scales, looking for locations that 
might contain a face (i.e.. looking for eye locations). The 

w detection system 43 then uses an arbitrator to combine 
Ihe filter outputs. The arbitrator is used to merge detec- 
tions from individual filters and eliminate overlapping 
detections. Using the neural network-based face detec- 
tion technology makes the face detection robust, rela- 

i5 tively fast and detect most faces. In addition, it allows 
the detection system 43 to detect different kinds of faces 
with different poses and lightings. 
[0034] Once a face is detected, the face detection and 
feature data extraction system 43 computes the face 

20 feature data of the detected face. In one embodiment, 
the face detection and feature extraction system 43 
computes the face feature data of the detected face 
using a face feature extraction technology described in 
the above-mentioned U.S. Pat. No. 5,164,992. Alterna- 

25 tively, other known face feature extraction schemes may 
be used. 

[0035] The face feature extraction process by the sys- 
tem 43 is done by first normalizing, warping (i.e., rotat- 
ing and scaling), and masking the face to produce a 

30 reference face that has a set of predetermined eye loca- 
tions. This means that through normalizing, warping, 
and masking of the detected face, the eyes of the 
detected face are mapped into the predetermined eye 
locations of the reference face. This is done by translat- 

35 ing, scaling, and rotating the detected face and by 
adjusting the brightness and contrast of the detected 
face, in other words, this is to standardize or normalize 
the detected face. This is disclosed in the above-men- 
tioned U.S. Pat No. 5,164,992. 

«o [0036] Then the system 43 projects the standardized 
face into an Eigen-face space to compute the face coef- 
ficients or face feature data of the detected face. This is 
also disclosed in the above-mentioned U.S. Pat. No. 
5.164,992. 

45 [0037] After the face detection and feature extraction 
system 43 extracts the face feature data, the extracted 
face feature data is then sent to the search engine 42. 
The search engine 42 then accesses the profile data- 
base 13 (Figure 1) with the extracted face feature data. 

so This is to find and locate any face feature data stored in 
the profile database 13 that matches or is similar to the 
extracted face feature data. 

[0038] The search engine 42 uses the extracted face 
feature data to search for face images stored in the pre- 
ss fie database 13 having similar face feature data. The 
search engine 42 compares the extracted face feature 
data with each stored face feature data to output similar- 
ity scores based on the predetermined similarity meas- 
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ure. In one embodiment, the similarity measure includes 
a set of thresholds and the face feature data includes a 
set of values. The comparison of the corresponding val- 
ues of two face feature data yields differences between 
the values of the two data. The differences are then 
compared against their corresponding thresholds. A 
similarity score is obtained based on how many of the 
differences are less or more than the thresholds. The 
search engine 42 can be any Kind of known search 
engine. 

[0039] The search engine 42 then retrieves the per- 
son-identifying data associated with the matching face 
feature data {i.e.. the similar faces} from the prof ie data- 
base 13. In one embodiment the search engine 42 
retrieves both the person-identifying data and the pixel 
data of the similar images for display after a search. In 
another embodiment, the search and engine 42 selec- 
tively retrieves the person-identifying data of the similar 
images in accordance with their similarity scores. 
[0040] When there is no matching face feature data in 
the profile database 1 3 and the input face image is to be 
stored in the profile database 13, the face detection and 
feature extraction system 43 extracts the face feature 
data from the input face image and associates the face 
feature data with the input face image before storing the 
image and the face feature data into the profile data- 
base 13. In addition, the face detection and feature 
extraction system 43 receives the person-identifying 
data of the input face image from the user interface 41 
before storing the pixel data and the face feature data of 
the input face image into the profile database 13. The 
person-identifying data of the input face image is also 
associated with the face feature data of the input face 
image in the profile database 13. 
[0041] Referring backto Figure 1, the person identifier 
and management system 1 o can be a portable personal 
system that can be easily earned around by the user of 
the system 10. For example, the person identifier and 
management system 10 can be a hand-held personal 
system. Alternatively, the person identifier and manage- 
ment system 10 can also be a non-portable system 
Figure 2 is a perspective view of a hand-held portable 
personal person identifier and management system 20 
that implements one embodiment of the person identi- 
fier and management system 10 of Figure 1 . Figures 3A 
and 3B show the front and top views of the hand-held 
portable personal person identifier and management 
system 20. 

[0042] As can be seen from Figures 2 and 3A-3B, the 
hand-held portable personal person identifier and man- 
agement system 20 includes a touch sensitive screen 
display 22 that allows display of the input face image 24 
and the retrieved person-identifying data 25 and a dig- 
ital camera 23 that is an integrated part in the hand-held 
portable personal person identifier and management 
system 20. The digital camera 23 is used to capture the 
input face image 24. In addition, the hand-held portable 
personal person identifier and management system 20 



also include a writing tool 21 that allows input into the 
hand-held portable personal person identifier and man- 
agement system 20 via the touch sensitive screen dis- 
play 22. 

5 [0043] In the foregoing specification, the invention has 
been described with reference to specific embodiments 
thereof. It will, however, be evident to those skilled in the 
art that various modifications and changes may be 
made thereto without departing from the scope of the 

w invention. The specification and drawings are. accord- 
ingly, to be regarded in an illustrative rather than a 
restrictive sense. 

Claims 

J5 

1. A person identifier and management system (10), 
comprising: 

(A) an image database (13) that stores face 
20 feature data of each of a plurality of face 

images and person-identifying data associated 
with each of the face images; 

(B) an image capturing system (11, 23) that 
captures an input face image: 

25 (C) a face analysis system (12) coupled to the 

image capturing system (1 1 , 23) and the image 
database (13) to locate and retrieve the per- 
son-identifying data of any stored face image 
similar to the input face image from the image 

so database (13). 

2. The person identifier and management system (10) 
of claim 1, wherein the face analysis system (12) 
locates and retrieves the person-identifying data 

35 from the image database (13) by extracting the fea- 
ture data of the input face image and comparing the 
face feature data of the input face image with that 
(101 a) of each of the face images stored in the face 
image database (13). 

40 

3. The person identifier and management system (1 0) 
of claim 1 or 2, wherein the person identifier and 
management system (10) is a portable personal 
person identifier and management system, wherein 

45 the face analysis system (1 2) further comprises 

(I) a face detection and feature extraction sys- 
tem (43) that detects the input face image and 
extracts the face feature data from the input 

so face image; 

(II) a search engine (42) that compares the 
face feature data of the input face image with 
that of the face images stored in the image 
database (13). 

55 

4. The person identifier and management system (10) 
of any of claims 1 -3, wherein the face analysis sys- 
tem (1 2) further comprises a user interface (41) that 
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allows the Input face image and the person-identify- 
ing data to be displayed on a display and that allows 
a user to input and associate the person-identifying 
data with a face image displayed. 

5 

5. The person identHier and management system (10) 
of claims 1 -4, wherein the image capturing system 
is an integrated digital camera. 

6. The person identifier and management system (10) to 
of claim 5. wherein the person-identifying data 
includes name, affiliation, and time and occasion of 
meetings. 

7. The person identifier and management system (10) is 
of claim 6. further comprising a display (22) that dis- 
plays the person-identifying data along with the 
input face image. 

8. The person identifier and management system (10) 20 
of claim 7, wherein the display (22) is a touch 
screen display that also allows input of the person- 
identifying data into the face image database (13). 
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